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ABSTRACT 



A system and method for creating multimedia information 
synchronized with a video signal (e.g., a program of a 
television broadcast) are provided. The multimedia infor- 
mation may be formatted in accordance with, for example, 
the MHEG standard or the ATVEF standard. The system 
includes means for specifying and verifying the contents of, 
and synchronizations for, the multimedia information. A 
scenario creation program creates a scenario composed of a 
plurality of scenes synchronized with playback of the video. 
A scene creation program combines objects consisting of 
text, images, audio, operation buttons and the like to create 
scenes (pages). A contents management program, a scenario 
management program and a scene management program set 
a directory of the contents, the scenarios and the scenes. 
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INFORMATION PROCESSING APPARATUS, preferably comprises means for identifying the video's 

METHOD, AND COMPUTER-READABLE starting time for displaying scenes of the multimedia infor- 

MEDIUM mation synchronized with such starting time. 

The information processing system preferably also com* 
S prises a display, and a display controller, for displaying a 
FIELD OF THE INVENTION time axis of the video and symbols corresponding to a 
The present invention relates to an information processing hier f arc ^ of , the scenes - 11,0 information processing system 
apparatus, method and computer-readable medium and, f. refe ' abl y also ""J™" mean * f ° r «0»W » synchronrza- 
more particularly, to an information processing apparatus, , n J 00 . fla f f ° r gating or deactivating synchronized play- 
method and computer-readable medium for creating, dis- 10 ba = k of lhe "idlnnedi* information with the video. A 
playing and synchronizing audio, pictorial and textual data ^id-processing playback flag for allowing or disallowing 
with video information. mid-processmg display of an image also preferably is pro- 

video. Disallowing such mid-pro cessmg display prevents 

BACKGROUND OF THE INVENTION the image's display to a viewer who begins viewing the 

15 video at a point when the image's display would be mean- 
Standards exist, or have been proposed, for displaying ingless. 
information and data, including moving images, still images, jhe information processing system preferably also corn- 
sound and text in response to a user's interactive commands, prises managing means for organizing, in a hierarchical 
and for synchronously displaying such information and data. manner, the multimedia information into contents, scenarios 
One such standard, proposed by the International Organiza- 20 and sc tncs for managing this hierarchical structure, 
tion for Standardization (ISO), is that defined by the Mul- Means for setting the pr i or i ty i eve i 0 f a scenej sucn that the 
timedia and Hypermedia Information Coding Experts Group rate at wmcn tne scene is transmitted from a broadcast 
(MHEG). Another standard is that defined by the Advanced station corresponds to a viewer's likelihood of selecting the 
Television Enhancement Forum (ATVEF), a consortium of scene for aisplay, also preferably is provided, 
broadcast and cable networks, consumer electronics 25 ^ information p rocessing apparatus preferably also 
companies, television transport operators and technology comprises means for Mthlg a &pUy mode flag for deter . 
companies. mining whether a viewer can control the multimedia infor- 
Information such as that corresponding to the MHEG mation's display. The setting of this flag prevents the viewer 
standard or the ATVEF standard (hereinafter referred to f rom switching off this display, 
generally as "multimedia information") can be provided to 30 

a viewer in a television broadcast. Other methods of trans- BRIEF DESCRIPTION OF THE DRAWINGS 

mission and display are contemplated, however, including pic. 1 is a block diagram of a system for broadcasting 

over the Internet to computers. multimedia information created using the apparatus and 

Multimedia information generally changes little with method of the present invention, 

respect to the video to which it corresponds. To provide 35 FIG. 2 is a block diagram of an apparatus for creating 

multimedia information at an appropriate time, however, the multimedia information in accordance with the present 

multimedia information should be synchronized with the invention. 

video to which it corresponds (for example, the moving FIG. 3 is a block diagram of the sections of a computer 

images of a television broadcast). An authoring system or program executed by me multimedia creating apparatus of 

authoring tool for creating such synchronized multimedia p\G 2 

information, however, does not exist. An object of the F j a 4 Ls an alustration of me objects of a scene . 

present invention is to provide such an authoring system or m „ m . .„ . „ „ 

authoring tool 5 is an illustration of a directory for the multimedia 

information showing the information's hierarchical struc- 

SUMMARY OF THE INVENTION 45 turc - 

_ . . , . , . FIG. 6 is a block diagram of IRD 4 of FIG. 1. 

The present invention provides an information processing r, Tr , * • li i j- c *i_ *• r 

j j . • i j c • FIG. 7 is a block diagram ot the sections of a computer 

system and method. The system includes means for speci- fc IRD 4 for { { ^ muhimcdia 

fying multimedia mformation and video with which the information 

multimedia information is synchronized, means for creating <n « • ' n • <- n 

the multimedia information and means for setting the timing u * 15 a . flowchart .^ratmg the steps for confirming 

for displaying scenes of the multimedia information to the synchronization ^ettings created by the multimedia ere- 

synchronize this information with the video. The method atlDg a PP aratus of HCr - 2 - 

includes specifying multimedia information and video with FIG * 9 15 an exam P le of a ^P 1 ^ Wiring on display 

which the multimedia information is to be synchronized, S5 a PP a ^s 15 of the multimedia ere atmg apparatus of FIG, 2 

creating the multimedia information and setting timing for pursuant to steps SI and S3 of FIG. 8. 

displaying scenes of the multimedia information to synchro- FIG - 10 is a flowchart illustrating the steps for confirming 

nize this information with the video. A computer-readable the interactive setting process created by the multimedia 

medium containing code for executing this method also is creating apparatus of FIG. 2. 

provided. In preferred embodiments, the multimedia infor- 6Q FIG. 11 is an illustration of the link structure of scenes, 

mation is either MHEG information or ATVEF information. FIG. 12 is an example of a display appearing on display 

Trie specifying means preferably comprises a contents apparatus 15 in accordance with the process of FIG. 10 for 

creating section for identifying the contents of the multime- confirming interactive settings. 

dia information and of the video with which the multimedia FIG. 13 is an illustration of the effect of the setting of 

information is to be synchronized. The creating means 65 synchronization flags. 

preferably comprises a scene creating section for creating FIG. 14 is an illustration of the effect of the setting of 

scenes of the multimedia information, and the setting means mid-processing display flags. 
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FIG. 15 illustrates a sending rate for scenes. 

FIG. 16 further illustrates a sending rate for scenes. 

DESCRIPTION OF THE PREFERRED 

EMBODIMENTS 5 

A broadcasting system for broadcasting video and multi- 
media information, such as, e.g., MHEG information or 
ATVEF information, is shown in FIG. 1. Broadcast station 
1 transmits television broadcasting signals containing mul- io 
tiplexed multimedia information signals. Antenna 3 receives 
these signals via satellite 2. The received signals are sup- 
plied to IRD (Integrated Receiver and Decoder) 4 which 
extracts the television and multimedia signals on a prede- 
termined channel. The extracted signals are supplied to 15 
display apparatus 5 for displaying the multimedia informa- 
tion superposed upon the video of a television broadcast. 

FIG. 2 is a block diagram of an apparatus in accordance 
with the present invention for creating multimedia informa- 
tion for transmission by broadcast station 1. The apparatus 20 
of FIG. 2 can be used to create any form of multimedia 
information, for example, MHEG information or ATVEF 
information. For the purpose of this example, however, it 
will be assumed that the apparatus of FIG. 2 is used to create 
MHEG information. 25 

The apparatus of FIG. 2 includes CPU 11 for providing 
overall control in accordance with a program stored in ROM 
12 and RAM 13. CPU 11 may be, e.g., a microprocessor or 
similar device. CPU 11 also executes a multimedia author- 3Q 
ing program stored on, e.g., hard disk 14 for creating MHEG 
information. The authoring program can, of course, be 
stored on any computer-readable medium accessible to CPU 
11, e.g., a floppy disk, magnetic tape, optical disk, etc. 

Display apparatus 15 displays the MHEG information, 35 
and the video to be synchronized with the MHEG 
information, to an operator. Keyboard 16 and mouse 17 
enable the operator to control the apparatus' operations. A 
communication section 18 including, for example, a modem, 
controls communications over a telephone line, the Internet, 40 
or the like. Sound playback apparatus 19 plays back sound 
to be synchronized with the video or MHEG information. 

FIG. 3 shows the functional blocks (sections) of the 
multimedia information authoring program executed by 
CPU 11. As indicated above, in this example, it is assumed 45 
that the authoring program is for authoring MHEG infor- 
mation. Although each of these blocks receives information 
from one or more of the other blocks and provides infor- 
mation to one or more of the other blocks, for clarity of 
presentation, arrows among the blocks indicating such 50 
inputs and outputs are omitted. 

A contents creation section 21 creates contents informa- 
tion for the MHEG information. This contents information 
corresponds to a video (e.g., a program of a television 
broadcast) to be synchronized with the MHEG information. 55 
The contents information is provided to the contents man- 
agement section 24. 

The contents information corresponds to a group of 
scenarios and comprises eight items of information: 6Q 

(1) contents_name 

(2) service_name 

(3) program__name 

(4) es_number 65 

(5) start_time 

(6) duration 



,800 Bl 

4 

(7) additional_info 

(8) application_number 

In the contents__name, "contents_1988_4_l_l," for 
example, is recorded to identify the name of the contents. In 

the service name, "NHK," for example, is recorded to 

identify the channel on which the program including the 
multimedia information is broadcast. In the program_name, 
"baseball," for example, is recorded to identify the pro- 
gram's name. In the es_number, "10" for example, is 
recorded to identify the number of elementary streams. In 
the start_time, "+00/00/00," for example, is recorded to 
identify the relative time for starting the display of the 
MHEG information with respect to the video with which the 
MHEG information is synchronized. In the duration, "00/ 
00/00," for example, is recorded to indicate the duration of 
the multimedia information's display from the relative start 
time. In the additional_info, "scenario 1," for example, is 
recorded to indicate that the information corresponds to the 
first scenario. In the application number, "3," for example, is 
recorded to indicate the number of scenarios contained in the 
contents. 

The contents creation section 21 also manages an appli- 
cation information file for recording the start time and 
duration for broadcasting a scene (described later with 
reference to FIG. 9). 

The scenario creation section 22 creates a scenario, and 
scenario information, and provides its output to the scenario 
management section 25. A scenario is composed of a plu- 
rality of scenes synchronized with the video. The scenario 
information comprises four items of information: 

(1) es__name 

(2) data__type 

(3) customized_jhfo 

(4) scene_jiumber 

In the es_number, "DATA 1," for example, is recorded to 
identify the name of the elementary stream corresponding to 
the scenario. In the data_type, "mheg," for example, is 
recorded to identify the type of multimedia information. In 
the customized_info, any specific information correspond- 
ing to the scenario may be recorded. In the scene_number, 
"3," for example, is recorded to indicate the number of 
scenes in the scenario. 

The scene creation section 23 combines "objects." As 
shown in FIG. 4, the objects make up a scene (page) and may 
be textual information, image information, audio 
information, operational buttons, and the like. The output of 
scene creation section 23 is provided to scene management 
section 26. 

A scene can be switched on and off either by synchroni- 
zation signals contained in the video signal or through a 
viewer's interactive operations, for example, activating 
operational buttons displayed in the scene. Further, it is 
possible to set, in the operational buttons, functions for 
executing predetermined processing, for example, to pro- 
vide textual information to be displayed for applying for the 
purchase of a product. 

A scene is composed of scene data and six items of 
information corresponding to these data: 

(1) scene_priority_level 

(2) flle__name 

(3) stream__event name 

(4) stream_event time 

(5) stream_name 

(6) es„name 

In the scene__priority_Ievel, "1," for example, is 
recorded to identify the sending rate (described later) for the 
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scene. In the file_name, "bitmap.jpg," for example, is 
recorded to identify the name of an object. In the stream_ 
event_name, "stream event 1," for example, is recorded to 

identify the name of the stream event. In the stream_event 

time, "00/00/00/' for example, is recorded to identify the 5 
starting time for sending the scene. In the stream_name, 
"stream link 1," for example, is recorded to identify the 
name of the stream. In the es name, "DATA 1", for example, 
is recorded to identify the name of the elementary stream of 
a link destination. 10 

The scene creation section 23 also manages scene infor- 
mation and records within this information synchronization 
flags (described below with reference to FIG. 13), mid- 
processing playback mode flags (described below with ref- 
erence to FIG. 14) and display mode flags (described 15 
below). 

The contents management section 24, the scenario man- 
agement section 25 and the scene management section 26 set 
the directory for the contents, the scenarios and the scenes. 
An example is shown in FIG. 5. In this example, a contents 20 
CI is set in the highest-order directory, and scenarios SCI, 
SC2 and SC3 are set in the next highest-order directory. In 
the scenario SCI, three scenes, scene 1, scene 2 and scene 
3, are set. In the scenarios SC2 and SC3, scene 4 and scene 
5, respectively, are set. 25 

Referring again to FIG. 3, a time control section 27 reads 
time information for the video to be played back by video 
playback section 29 and outputs this information to a video 
and MHEG display control section 31. An interactive con- 
trol section 28 records settings for switching the display of 30 
the scenes in response to operations of the user. 

The video playback section 29 plays back the video with 
which the MHEG information is to be synchronized. The 
MHEG playback section 30 plays back the MHEG infor- 
mation to verify that the creation, synchronizations, etc. are 35 
correct. The video and MHEG display control section 31 
controls the display apparatus 15 (FIG. 2) so that the MHEG 
information played back by the MHEG playback section 30 
is displayed superposed over the video information played 
back by the video playback section 29. 40 

FIG. 6 shows the construction of IRD 4 (FIG. 1) for 
playing back broadcasted video and multimedia information 
to a viewer. A receiving processing section 41 containing a 
down converter, a tuner, a demodulator, a descrambler and 
the like, extracts predetermined channel components 45 
(television broadcasting signals and MHEG information 
signals) from the broadcasted signals received on antenna 3 
and supplies these components to CPU 42. 

CPU 42 controls IRD 4 through execution of a control 
program, stored in ROM 43 and RAM 44, and a MHEG 50 
information playback program stored on hard disk 45. The 
broadcasted video and MHEG information are displayed on 
display apparatus 5 (FIG. 1). 

A display control section 46 controls superposing of the 
MHEG information over the broadcasted video displayed on 55 
display apparatus 5. Keyboard 47, mouse 48, and remote 
controller 49 provide input from a viewer. A communication 
section 50, for example, a modem, controls information 
communications over a telephone line, the Internet and the 
like. 60 

FIG. 7 shows the functional blocks of the MHEG infor- 
mation playback program for IRD 4 stored on hard disk 45 
and executed by CPU 42. A video playback section 54 plays 
back the video of a channel selected by the viewer. A MHEG 
playback section 55 plays back MHEG information corre- 65 
sponding to this video. A video display control section 56 
controls the display apparatus 5 (FIG. 1). This control causes 



the MHEG information from MHEG playback section 55 to 
be superposed over the video from video playback section 
54. 

A directory management section 51 manages the MHEG 
information's directory and provides this information to the 
MHEG playback section 55 in accordance with predeter- 
mined timing. A timing control section 52 performs time 
measurement using, as a reference, the time at which the 
video playback section 54 starts playback of the video 
corresponding to the MHEG information and informs the 
MHEG playback section 55 of, for example, the start_time 
contained in the contents information. 

An interactive control section 53 detects operations by a 
viewer upon the keyboard 47, the mouse 48 or the remote 
controller 49, and performs predetermined processing pur- 
suant to these operations, such as switching a scene. 

Referring again to the apparatus for creating multimedia 
information shown in FIG. 2, by operating the keyboard 16 
(FIG. 2) and setting predetermined values in the contents 
information (e.g., start_time and duration), a scene can be 
set to be interactively synchronized with the video defined in 
program_name. These settings then can be confirmed by 
CPU 11 in response to an operator's instructions received 
from keyboard 16 or mouse 17. In response, CPU 11 
performs the processing shown in the flowchart of FIG. 8. 

As shown in this figure, in step SI, the time control 
section 27 (FIG. 3) reads the time information (start time and 
end time) of the video to be played back by video playback 
section 29. This time information is provided to the video 
and MHEG display control section 31. As shown in FIG. 9, 
the video and MHEG display control section 31 causes 
display apparatus 15 to display a time axis for the video 
based upon the input time information. 

In step S2, the time control section 27 (FIG. 3) reads the 
playback start time (start_time) and the playback duration 
(duration) for a scenario to be synchronized with the video. 
This information also is provided to video and MHEG 
display control section 31. The operator also provides the 
sending start time (time that broadcasting begins) and the 
sending duration (duration of broadcasting) for the MHEG 
data. This information is recorded in the application infor- 
mation file of the contents creation section 21. The time 
control section 27 also reads this information and provides 
it to the video and MHEG display control section 31. 

In step S3, the video and MHEG display control section 
31 causes display apparatus 15 to display the playback start 
time, the playback duration, the sending start time and the 
sending duration along the time axis displayed in step SI. 
This display, shown in FIG. 9, enables the operator to 
visualize, and confirm, the synchronization settings for the 
MHEG information. 

In the example shown in FIG. 9, the playback start times, 
the playback durations, the sending start times and the 
sending durations for the scenarios SCI and SC2 are set with 
respect to the start time for the corresponding video pro- 
gram. For scenario SCI, in this example, the playback start 
time is set for the same time as the program start time, the 
playback duration is set for 10 minutes, the sending start 
time is set for 5 minutes before the program start time and 
the sending duration is set for 20 minutes. For scenario SC2, 
the playback start time is set for 20 minutes after the 
program start time, the playback duration is set for 10 
minutes, the sending start time is set for 15 minutes after the 
program start time, and the sending duration is set for 20 
minutes. 

During the 5 minute durations after completion of play- 
back of the scenarios, a viewer can, for example, make a 
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telephone call to purchase an item that was displayed (in the selections for display after scene 2) in the same hierarchy as 

case of a shopping program). Since the MHEG creating that for scene 6. 

apparatus enables recording of the settings such that the In step S28, the interactive control section 28 determines 

MHEG information is transmitted only during the necessary whether the symbol for scene 6 has been selected. If this 

times, the transmission channel is used effectively for trans- 5 section determines that the symbol for scene 6 has been 

mitting a plurality of MHEG data in a time divisional selected, the process proceeds to step S29 where the video 

manner. and MHEG display control section 31 pulls down the 

The process of confirming the interactive settings for the symbols for scenes 7 and 8 and displays these symbols 

MHEG information prepared by the creation apparatus of below the symbol for scene 6. 

FIG. 2 is described with reference to FIGS. 10, 11 and 12. 10 In step S30, the interactive control section 28 determines 

This process enables the user to visualize, and confirm the whether one of the symbols for scenes 6 through 8 has been 

accuracy of, the scenes' interactive relationships with selected. If this section determines that one of these symbols 

respect to time (link relationships) after the synchronization has been selected, the process proceeds to step S31. In this 

settings, such as playback start time and the like, have been example, it is assumed that the symbol for scene 6 has been 

set and the scenes within the scenario have been linked. As 15 selected. 

an example, as shown in FIG. 11, assume that scenes 2 In step S31, the video and MHEG display control section 

through 5 are set as possible selections for display by the 31 displays the symbol for scene 6 below that for scene 2. 

viewer after scene 1 and that scenes 6 through 8 are set as Since no scenes are present in a hierarchy below that for 

possible selections for display by the viewer after scene 2. scene 6, the processing terminates. 

Upon receipt of an instruction for starting the confirma- 20 The contents creation section 21 (FIG. 3) also enables a 

tion process on keyboard 16 or mouse 17 (FIG. 2), the time user to set synchronization signals (flags) within the stream 

control section 27 (FIG. 3) reads, in step S21 (FIG. 10), the of video information. These flags are set in the scene 

time information (start time and end time) for the video to information file by the scene creation section 23 and are 

be played back by the video playback section 29 (FIG. 3) processed by IRD 4 (FIG. 1). The setting of a synchroniza- 

and supplies this information to the video and MHEG 25 tion flag for a scene to the on state causes the scene's 

display control section 31 (FIG. 3). As shown in FIG. 12, the switching to be synchronized with the video. The setting of 

video and MHEG display control section 31 causes display such a flag to the off state disables such synchronization with 

apparatus 15 (FIG. 2) to display a time axis for the video the video. 

based upon the received time information. For example, as shown in FIG. 13, assume that scenes 1 

In step S22, the time control section 27 next reads the 30 through 4 form scenario SCI and a synchronization flag for 
playback start time and the playback duration for the first switching to scene 3 in the middle of scene 2 is received 
scene (scene 1) and supplies this information to the video from the video signal. Assume further that in pattern 1, the 
and MHEG display control section 31. In step S23, as shown synchronization flag is off and in pattern 2, the synchroni- 
in FIG. 12, the video and MHEG display control section 31 zation flag is on. As shown in FIG. 13, in pattern 1 
causes display apparatus 15 to display, below the time axis 35 (synchronization flag off), after scene 2 is terminated, scene 
for the video, a symbol indicating the playback duration for 3 is played back (the playback is not synchronized with the 
scene 1. This symbol starts 10 minutes after the video start video). In pattern 2 (synchronization flag on), although 
time. The video and MHEG display control section 31 also playback of scene 2 is incomplete, the scene nevertheless is 
causes display apparatus 15 to display a symbol for scene 2 switched to scene 3 (synchronized playback is performed), 
adjacent to the symbol for scene 1. The video and MHEG 40 The scene creation section 23 also may set a mid- 
display control section 31 further causes display apparatus processing playback mode flag in the scene information file. 
15 to display a triangular mark within the symbol for scene For example, as shown in FIG. 14, assume that scenes 1 
2 to indicate that other scenes are present (possible other through 3 form scenario SCI, that these scenes are set to be 
selections for display after scene 1) in the same hierarchy as played back in sequence and that a viewer's playback of the 
that for scene 2. 45 video signals begins during mid -processing of scene 2. 

In step S24, the interactive control section 28 determines Assume further that, in pattern 3, the mid-processing play- 

whether the user has selected the symbol for scene 2 (e.g., back mode flag is set to the on state and in pattern 4, the 

clicked by the mouse 17). If this section determines that the mid-processing playback mode flag is set to the off state. As 

symbol for scene 2 has been selected, the process proceeds shown in FIG. 14, in pattern 3, the playback of scene 2 is 

to step S25. 50 started during the mid -processing of this scene (since the 

In step S25, the video and MHEG display control section mid-processing playback mode flag is on, a mid-processing 

31 pulls down the symbols for scenes 3 through 5, as shown display occurs). In pattern 4, on the other hand, when 

in FIG. 12, and causes display apparatus 15 to display these playback of the video signals begins (for example, IRD 4 is 

symbols below the symbol for scene 2. turned on and starts to decode received data), the play back 

In step S26, the interactive control section 28 determines 55 of scene 2 does not begin even if scene data are transmitted 

whether the user has selected one of the symbols for scenes (since the mid-processing playback mode flag is off, a 

2 through 5. If this section determines that one of these mid-processing display does not occur). The first scene 

symbols has been selected, the process proceeds to step S27. played back is scene 3 which begins after termination of the 

In this example, it is assumed that the symbol for scene 2 has time for playback of scene 2 and at the time set for starting 

been selected. 60 play back of scene 3. This flag prevents the display of 

In step S27, as shown in FIG. 12, the video and MHEG multimedia information which is meaningful only if dis- 

display control section 31 causes the display apparatus 15 to played in synchronization with the video, 

display the symbol for scene 2 below the symbol for scene The scene creation section 23 also can set a display mode 

1 and further display the symbol for scene 6, which is linked flag for transmission to IRD 4 to disable a viewer's ability 

to scene 2, in a symbol adjacent to the symbol for scene 2. 65 to turn off multimedia information otherwise selectable by 

Within the symbol for scene 6, a triangular mark is displayed the viewer. For example, this flag may be used if the scene 

to indicate that other scenes are present (possible other is a commercial synchronized with a commercial video and 
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display of the multimedia information on display apparatus 
5 is desired regardless of the viewer's intent. If the display 
mode flag is set to the on state, the scene is displayed 
regardless of the viewer's selection on the controls for IRD 
4. If the display mode flag is set to the off state, the viewer's 
selection on the controls for IRD 4 determines whether the 
MHEG information is displayed. 

The scene creation section 23 also can set, in the scene__ 
priority level for the scene, a number to determine the rate 
at which the scene is transmitted from broadcasting system 
1. For example, as shown in FIG. 15, if scenes 1 through 3 
form scenario SCI and scene 1 (the initial scene) is selected 
for display most frequently by viewers, the sending rate for 
scene 1 can be set to, e.g., "2" and the sending rate for scenes 
2 and 3 can be set to, e.g., "1". As shown in FIG. 16, in the 
information stream for scenario SCI whose transmission 
rate is set in this manner, the information for scene 1 is sent 
twice as frequently as that for scenes 2 and 3. This trans- 
mission rate decreases the time before the most frequently 
viewed scene (scene 1) is displayed to the viewer upon 
selection. Although a ratio of integers is illustrated for the 
sending rate, this setting could be any ratio. 

Although the synchronization settings for the MHEG 
information with respect to the video are described above as 
being set using relative time, these synchronization settings 
also could be set with respect to real or absolute time. 

A computer program for performing the above-described 
methods may be stored on any computer-readable medium, 
such as an optical disk, magnetic tape or cd-rom, and such 
a program may be distributed through any means of distri- 
bution such as the Internet or digital satellites. 

What is claimed is: 

1. An information processing apparatus for creating mul- 
timedia information to accompany video information trans- 
mitted for display to a viewer, the information ocessing 
apparatus comprising: 

creating means for creating the multimedia information; 

identifying means for identifying video information with 
which the multimedia information is to be synchro- 
nized; 

setting means for setting a timing for displaying one or 
more scenes of the multimedia information so as to 
cause the displaying of the one or more scenes to be 
synchronized with the display of the video information 
to the viewer; and 

deactivation means for discriminately deactivating the 
synchronization of at least one scene of the multimedia 
information with the display of the video information 
while allowing synchronization of other scenes of the 
multimedia information. 

2. An information processing apparatus as in claim 1, 
wherein the multimedia information conforms to a Multi- 
media and Hypermedia Information Coding Experts Group 
(MHEG) standard. 

3. An information processing apparatus as in claim 1, 
wherein the multimedia information conforms to an 
Advanced Television Enhancement Forum (ATVEF) stan- 
dard. 

4. An information processing apparatus as in claim 1, 
wherein said setting means comprises means for setting the 
timing with respect to a relative time for the display of the 
video information. 

5. An information processing apparatus as in claim 1, 
further comprising: 

time-axis display means for displaying an axis corre- 
sponding to a time for the display of the video infor- 
mation; and 
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display control means for displaying, relative to the axis, 
symbols corresponding to the timing for displaying the 
one or more scenes. 

6. An information processing apparatus as in claim 1, 
further comprising mid-processing means for setting a flag 
to prevent display of a scene of the multimedia information 
whose playback is synchronized to begin at a point in the 
video information which occurs before a point at which the 
viewer begins playing back the video information. 

7. An information processing apparatus as in claim 1, 
further comprising management means for organizing and 
managing the multimedia information in a hierarchy of 
contents, scenarios and scenes. 

8. An information processing apparatus as in claim 1, 
further comprising transmission rate setting means for set- 
ting a rate at which a plurality of the scenes are transmitted 
to the viewer. 

9. An information processing apparatus as in claim 1, 
further comprising display disabling means for setting a flag 
to disable the viewer's ability to prevent the display of the 
multimedia information. 

10. An information processing method for creating mul- 
timedia information to accompany video information trans- 
mitted for display to a viewer, the information processing 
method comprising: 

creating the multimedia information; 

identifying video information with which the multimedia 
information is to be synchronized; 

setting the timing for displaying one or more scenes of the 
multimedia information so as to cause the displaying of 
the one or more scenes to be synchronized with the 
displaying of the video information to the viewer; 

setting a flag for the multimedia information for a^crirni- 
nately deactivating the synchronization of at least one 
scene of the multimedia information with the display of 
the video information, and 

setting flags for the multimedia information to allow 
synchronization of other scenes while the at least one 
scene is deactivated. 

11. An information processing method as in claim 10, 
wherein the multimedia information conforms to a Multi- 
media and Hypermedia Information Coding Experts Group 
(MHEG) standard. 

12. An information processing method as in claim 10, 
wherein the multimedia information conforms to an 
Advanced Television Enhancement Forum (ATVEF) stan- 
dard. 

13. An information processing method as in claim 10, 
wherein said setting a timing step includes the step of setting 
the timing with respect to a relative time for the display of 
the video information. 

14. An information processing method as in claim 10, 
further comprising the steps of: 

displaying an axis corresponding to a time for the display 

of the video information; and 
displaying, relative to the axis, symbols corresponding to 

the timing for displaying the one or more scenes. 

15. An information processing method as in claim 10, 
further comprising the step of setting a flag to prevent 
display of a scene of the multimedia information whose 
playback is synchronized to begin at a point in the video 
information which occurs before a point at which the viewer 
begins playing back the video information. 

16. An information processing method as in claim 10, 
further comprising the step of organizing and managing the 
multimedia information in a hierarchy of contents, 
scenarios, and scenes. 
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17. An information processing method as in claim 10, 
further comprising the step of setting a rate at which a 
plurality of the scenes are transmitted to the viewer. 

18. An information processing method as in claim 10, 
further comprising the step of setting a flag to disable the 
viewer's ability to prevent the display of the multimedia 
information. 

19. A computer- readable medium having computer- 
executable instructions stored thereon for creating multime- 
dia information to accompany video information transmitted 
for display to a viewer, the computer-executable instructions 
when executed by a computer causing the computer to 
perform a method comprising the steps of: 

creating the multimedia information; 

identifying video information with which the multimedia 
information is to be synchronized; 

setting the timing for displaying one or more scenes of the 
multimedia information so as to cause the displaying of 
the one or more scenes to be synchronized with the 
display of the video information to the viewer; 

setting a flag for the multimedia information for discrimi- 
nately deactivating the synchronization of at least one 
scene of the multimedia information with the display of 
the video information; and 

setting flags for the multimedia information to allow 
synchronization of other scenes while the at least one 
scene is deactivated. 

20. A computer-readable medium as in claim 19, wherein 
the multimedia information conforms to a Multimedia and 
Hypermedia Information Coding Experts Group (MHEG) 
standard. 

21. A computer-readable medium as in claim 19, wherein 
the multimedia information conforms to an Advanced Tele- 
vision Enhancement Forum (ATVEF) standard. 

22. A computer-readable medium as in claim 19, wherein 
the setting a timing step includes the step of setting the 
timing with respect to a relative time for the display of the 
video information. 

23. A computer-readable medium as in claim 19, wherein 
the method further comprises the steps of: 

displaying an axis corresponding to a time for the display 

of the video information; and 
displaying, relative to the axis, symbols corresponding to 

the timing for displaying the one or more scenes. 

24. A computer- read able medium as in claim 19, wherein 
the method further comprises the step of setting a flag to 
prevent display of a scene of the multimedia information 
whose playback is synchronized to begin at a point in the 
video information which occurs before a point at which the 
viewer begins playing back the video information. 

25. A computer- read able medium as in claim 19, wherein 
the method further comprises the step of organizing and 
managing the multimedia information in a hierarchy of 
contents, scenarios and scenes. 

26. A computer- readable medium as in claim 19, wherein 
the method further comprises the step of setting a rate at 
which a plurality of the scenes are transmitted to the viewer. 

27. A computer- readable medium as in claim 19, wherein 
the method further comprises the step of setting a flag to 
disable the viewer's ability to prevent the display of the 
multimedia information. 

28. An apparatus for synchronizing display of a video 
with multimedia information corresponding to the video, the 
apparatus comprising: 

means for specifying a duration for displaying the video; 

means for specifying a period for displaying each scene of 
a plurality of scenes of the multimedia information with 
respect to the duration for displaying the video; 
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means for displaying on a computer screen a plurality of 
symbols, each of the symbols corresponding to one of 
the scenes and representing a relative time and position, 
with respect to the duration for displaying the video, of 
a period corresponding to the scene; and 

means for setting a flag to prevent display of a scene of the 
multimedia information whose playback is synchro- 
nized to begin, with respect to the video, at a point in 
the video which occurs before a point at which a viewer 
begins playing back the video; 

whereby a user is capable of distinguishing which scene 
and which sequence of scenes to display within the 
duration of the broadcasting of the video. 

29. An apparatus as in claim 28, further comprising means 
for displaying on the computer screen, concurrently with the 
symbols, a second symbol representing the duration for 
displaying the video. 

30. An apparatus as in claim 28, wherein the multimedia 
information conforms to a Multimedia and Hypermedia 
Information Coding Experts Group (MHEG) standard. 

31. An apparatus as in claim 28, wherein the multimedia 
information conforms to an Advanced Television Enhance- 
ment Forum (ATVEF) standard. 

32. An apparatus for synchronizing a video with multi- 
media information for broadcast, the apparatus comprising: 

means for specifying a duration for broadcasting the 
video; 

means for specifying a period for broadcasting each scene 
of a plurality of scenes of the multimedia information 
with respect to the duration for broadcasting the video; 

means for displaying on a computer screen a plurality of 
symbols, each of the symbols corresponding to one of 
the scenes and representing a relative time and position, 
with respect to the duration for broadcasting the video, 
of a period corresponding to the scene; and 

means for setting a rate at which a plurality of the scenes 
are transmitted to a viewer; 

whereby a user is capable of distinguishing which scene 
and which sequence of scenes to display within the 
duration of the broadcasting of the video. 

33. An apparatus as in claim 32, further comprising means 
for displaying on the computer screen, concurrently with the 
symbols, a second symbol representing a broadcasting dura- 
tion of the video. 

34. An apparatus as in claim 32, wherein the multimedia 
information conforms to a Multimedia and Hypermedia 
Information Coding Experts Group (MHEG) standard. 

35. An apparatus as in claim 32, wherein the multimedia 
information conforms to an Advanced Television Enhance- 
ment Forum (ATVEF) standard. 

36. A computer-readable medium having computer- 
executable instructions stored thereon for synchronizing 
display of a video with multimedia information correspond- 
ing to the video, the computer-executable instructions when 
executed by a computer causing the computer to perform a 
method comprising the steps of: 

specifying a duration for displaying the video; 

specifying a period for displaying each scene of a plural- 
ity of scenes of the multimedia information with 
respect to the duration for displaying the video; 

displaying on a computer screen a plurality of symbols, 
each of the symbols corresponding to one of the scenes 

setting a flag to prevent display of a scene of the multi- 
media information whose playback is synchronized to 
begin at a point in the video which occurs before a point 
at which a viewer begins playing back the video; 
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whereby a user is capable of distinguishing which scene 
and which sequence of scenes to display within the 
duration of the broadcasting of the video. 

37. A computer- readable medium as in claim 36, wherein 
the method further comprises the step of displaying on the 5 
computer screen, concurrently with the symbols, a second 
symbol representing the duration for displaying the video. 

38. A computer- readable medium as in claim 36, wherein 
the multimedia information conforms to a Multimedia and 
Hypermedia Information Coding Experts Group (MHEG) 10 
standard. 

39. A computer-readable medium as in claim 36, wherein 
the multimedia information conforms to an Advanced Tele- 
vision Enhancement Forum (ATVEF) standard. 

40. A computer-readable medium having computer- 15 
executable instructions stored thereon for synchronizing a 
video with multimedia information for broadcast, the 
computer-executable instructions when executed by a com- 
puter causing the computer to perform a method comprising 
the steps of: 20 

specifying a duration for broadcasting the video; 

specifying a period for broadcasting each scene of a 
plurality of scenes of the multimedia information with 
respect to the duration for broadcasting the video; 



displaying on the computer screen a plurality of symbols, 
each of the symbols corresponding to one of the scenes 
and representing a relative time and position, with 
respect to the duration for broadcasting the video, of a 
period corresponding to the scene; and 

setting a rate at which a plurality of the scenes are 
transmitted to a viewer; 

whereby a user is capable of distinguishing which scene 
and which sequence of scenes to display within the 
duration of the broadcasting of the video. 

41. A computer-readable medium as in claim 40, wherein 
the method further comprises the step of displaying on the 
computer screen, concurrently with the symbols, a second 
symbol representing a broadcasting duration of the video. 

42. A computer-readable medium as in claim 40, wherein 
the multimedia information conforms to a Multimedia and 
Hypermedia Information Coding Experts Group (MHEG) 
standard. 

43. A computer-readable medium as in claim 40, wherein 
the multimedia information conforms to an Advanced Tele- 
vision Enhancement Forum (ATVEF) standard. 
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